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The source of knowledge is experience. 

Experience is of two kinds-our own, & that of others. 
Knowledge gained from our own experience is most expensive. 
Mystery & ignorance are always coexistent. 

Learn the truth & the mystery will disappear. 



THE FEDERAL BUDGET 

Fiscal Year 2011 


National Aeronautics & Space Administration 

NASA’s Constellation program - based largely on 
existing technologies - was based on a vision of returning 
astronauts back to the Moon by 2020. 

However, the program was over budget , behind 
schedule, & lacking in innovation due to a failure to 
invest in critical new technologies. 

The President’s Budget cancels Constellation & 
replaces it with a bold new approach that invests in the 
building blocks of a more capable approach to space. 

In the last 20 years NASA has spent at least $21 B on 
canceled Space Transportation Programs 

~7% of its Budget 

Overruns can have real consequences 



National Aeronautics and Space Administration 




Mars Science Lab (MSL) Status 
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MSL Project Overview 



Science. 

Focus on Past & Present Habitability of Mars 

Highly Capable Analytical Laboratory 

Next Generation Remote Sensing & Contact Investigations 

Suite of Environmental Monitoring Instruments 



Planned Launch: Oct-Dec 2011 



Technical Capabilities 
One Mars Year surface operational 
lifetime (669 sols/687 days) 

Discovery Responsive over wide range of 
latitudes and altitudes 
Precision Landing via Guided Entry 
Skycrane Propulsive Landing 
Long Distance Traverse Capability (20 km) 
Flexible & Robust Sample Acquisition & 
Processing 
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July 2009 


Rebaselines 


$650M MSL Growth Timeline 





$1,381M 

2005 



$1,546M 

12/2005 



$1,645M 

8/2006 



$1,645M 

6/2007 


KDP accelerated up from 
Baseline 




Descope 

Review 




$1,671M 

8/2007 


□ One instrument 
Deleted & two 
instruments 



$1,886M 

jB Baseline 

6/2008 

1 0/2007 AA 


reinstates w 

Instrument 

~$200M 

Descopes H 

increase 



$2,031 M 
10/2008 


□ ~$140M 

increase 


have descopes 

□ ~$27M in 

descopes in 
addition to cost 
increase 


Rebaseline Review 06/2009 


Cost: $2,394M 
2011 Cost $2,470M 



$2,271 M 
12/2008 


□ ~$240M 
increase 

□ LRD delayed 
from 2009 to 
2011 


Current Launch date late 2011 
Original plan to launch in 2009 
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MSL Finding #4 


□ 


Lack of quantitative evidence for findings provided at KDP & 
Rebaseline Reviews 




Findings - Schedule 


Wirz StienC tf LibCHBlCM 


Same 
statement 
shown at both 
6/2008 and 
6/2009 Review 


Schedule has healthy margins 

- 3 month ealy delivery to launch site part of the plan 

- Plan includes completion of a I flight 5/W builds before launcli 

- 2 launch windows offer potential additional schedule relief (each has 
different implications for cove'age during EDL) 

• Type II Oct 15-Nov 18, Type I Nov 25-Dec 18 

- Integrated cosVschedule risk analysis shows LRD at early opening of 
window has a 97% or aetter probability, assuming adequate and timely 
budget is made available 

Avionics system (including hardware and software), actuators, rover 
power syslem, SAM Instrument represent a significant threat still 


• Project evaluating options for ATLO flow. SRB believes that options 
beyond the baseline "option 2” add technical and programmatic risk 




James Web Space Telescope (JWST) 
Cost & Schedule History 


6,000 

5.000 

4.000 
g 3,000 

2.000 
1,000 
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Current $6.5B 



FY95 FY99 FY01 FY03 FY04 FY06 FY09 


Current launch date rumored to be between 2018-2024 



Cost Growth 


History on 40 Spacecraft Projects 
Interrelationship of Cost and Schedule* 



200 % 

150% 

100 % 

50% 

0% 

-50% 

0% 20% 40% 60% 80% 100% 

Interrelationship of Cost and Schedule Growth for 
Non -Restricted Launch Window Projects 


Spacecraft Rule of Thumb: For every % Schedule 
Growth leads to equal or greater % of Cost Growth 


%Cost Growth = 1 .2348 * %Schedule Growth 
R 2 = 0.6124 

♦ 



T 


•Recent results from study of 40 
NASA missions show correlation 
between cost & schedule growth 

•The corresponding relationship 
states that for every 1 0 percent 
of schedule growth, there should 
be a corresponding 12 percent 
increase in cost growth. 

•Although the variations can be 
significant, a general rule of 
thumb could be followed that 
states that for every percent of 
schedule growth, there will more 
than likely be an equal or greater 
percent of cost growth. 


* Using Historical NASA Cost and Schedule Growth to Set Future Program and Project Reserve Guidelines 
Presented at the IEEE Aerospace Conference, March 3-10, 2007, Big Sky, Montana 




History of 96 Spacecraft Projects 

1st Available Cost Estimate to Last Available Cost Estimate 



Data includes projects 
at various phases of 
development. 


RiskBetaGeneral(0.98679, 26.007,-0.16733,29.928) 
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Cost Growth in Hundreds of % 


LO 


— Input 

Minimum 

-0.1673 

Maximum 

5.4619 

Mean 

0.9326 

Mode 

«0.0660 

Median 

0.5085 

Std Dev 

1.0726 

Skewness 

1.6606 

25% 

0.1519 

75% 

1.3469 

90% 

2.6022 

— BetaGeneral 

Minimum 

-0.1673 

Maximum 

29.9280 

Mean 

0.9328 

Mode 

-0.1673 

Median 

0.6100 

Std Dev 

1.0675 

Skewness 

1.8026 

25% 

0.1551 

75% 

1.3753 

i 90% 

2.3589 


Only 12.5% chance of completing project on or under budget. 

Average cost growth: 93% & Median: 51% 

If overruns are consistent they can be predicted 
Probably understated due to re-baselining & cancellation events. 





The project team protests.... 
“But these are spacecraft 

projects 

facilities are easier.” 
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Recent Facility Cost and Schedule Growth 



♦ 


♦ 


♦ 


♦ 


A-3 Test Stand 

♦ $157 M initial estimate 

♦ $320 M current project estimate 

♦ 2.04 times original 

♦ $366 M ICE estimate 

♦ Latest available duration 7 years 1. 75 times longer than original estimate 

SET 

♦ $ 63 M original estimate 

♦ $152 M latest available estimate 

♦ 2.41 times original estimate 

♦ Latest available duration 7 years 1. 75 times longer than original estimate 

O&C IOZ 

♦ $ 18 M original estimate 

♦ $55.2 M latest available estimate 

♦ $55.3 M ICE estimate 

♦ 3.07 time original estimate 

♦ Construction Duration 1 8 months original - actual 27.3 months (1.52 times) 

Hydro Impact Basin 

♦ Conceptual Design Construction Cost Estimate: $1.89M 

♦ Final Design Construction Cost Estimate: $2.60M 

♦ Construction duration 

♦ 90 days estimate 

♦ 210 days reality 

♦ 2.33 times longer than original 


Data 

suggests the 
larger the 
project the 
higher the 
cost growth 




•300 feet tall 
•Open steel frame 
structure 
•Up to 1 million 
pounds of thrust 
•Simulates altitudes 
up to 100,000 feet by 
generating steam to 
create a vacuum 


The new A-3 Test Stand will allow engineers to test operating 
parameters of the J-2X engine for the Ares launch vehicles by 
simulating conditions at different altitudes. 


A-3 Test Stand 




A3 Cost History 
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A3 - Schedule 


Independent 

Assessment 




Construction Test Need Date 

Activation 

Contingency 
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VAB Door Refurbishment 
A Tale of Two Projects 



1 

TOWER C 


| 

TOWER F 

|HB - 4 




HB-3 

TOWER B 


TOWER E 

|HB - 2 




HB-1 

TOWER A 


TOWER D 


i i 



□ y 2 building done cost plus 

□ y 2 building done fixed price 

□ Which one cost more? 



Transfer Aisle Doors 



6/21/2011 






Other Recent Facility Projects 
Cost and Schedule Growth 



□ Cost Plus VAB Door Renovation 

♦ $ 9.5 M original estimate 

♦ $56.5 M final cost 


♦ 5.95 times original estimate Essentially the same scope 


□ Fixed Price VAB Door Renovation done by the same contractor 


♦ $27.0 M original estimate 

♦ $27.6 M final cost 

♦ 1 .02 times original estimate — - 

□ Cost Plus Processing Facility 

♦ $135 M original estimate 

♦ $135M @ 60% design complete 

♦ $163M @ 90% design complete ± 5% 

♦ $180M @100% design complete 

♦ $370 M current estimate 

♦ 2.74 times original estimate 

♦ Scheduled duration 8. 6 years 1. 75 times longer than original estimate 
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Ok there may be some risk.... 
What are the causes? 
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My Personal Assessment 
Root Cause Analysis 


1 . Deliberate lowball estimates & optimistic assumptions 

□ Often with management “encouragement” to get project approved 

□ Once approved, and billions are spent, it's difficult to cancel project 

□ Projects repeatedly understate required reserves 

Despite history of overruns 

What is there is often transferred to other overrunning projects 
When problems arise, money is "borrowed" from next task 
Cash eventually runs out and, if more is not available, work gets delayed 
Problems compound and project costs balloon 

2. NaTve 

□ Project optimistic about risks & ability to defeat technical challenges 

Costs & schedules are based on best-case scenario and ignores risk impacts 

□ Project always assume they will quickly overcome challenges 

Problems that delay one task have ripple effect on next task, and the next one 

□ Delays have associated cost impacts, such as keeping workers while awaiting resolution of 
big problems. 

3. Engineers want to build perfect systems 

□ Big difference in costs, and “good enough” is all that is required 

Gets complex, and management cant or wont argue details 

4. Ignore/influence independent assessments 
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Example of Assumption Problems 




Shuttle Operations Concept 1974 
What was estimated 


OPF Today 
What was built 
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What is Good Enough? 


□ Common for engineers to “gold plate” specifications 

□ They want a perfect system - This is safe 

□ Example we built a 2,400 SF (223 SM) storage facility 

□ Engineers specified $18,000 electric panel when a $480 panel is 
more than adequate 

□ This is not a critical facility - Its for boat storage 
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Early Design Choices 
Affect Ability to Influence Costs 


High 



Low 

ATP 


■■ » 

Time 


Start 

Operations 




Rule of Thumb - The earlier a decision/change is made the lower the impact j 

will be to schedule and budget and money 
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Planning Stage 

Budget is $100M so estimate has 
to be $100M or less 

■ Often driver of other issues 

■ Belief problems will be easily overcome 

Optimistic Assumptions 

■ Money will show up on time 

■ We can use previous design from 
another project “as is” 

■ Escalation will be low 

Lack of accountability 

■ Historically no real repercussions for 
overrun, but rewards for starting 

■ Desire for perfection 

We have to “hurry up” and award 
contracts to meet schedule 

■ Initial schedule often politically set 

Lack of adequate upfront planning 


Implementation Stage 

■ Deny existence of any problems as 
long as possible 

■ Initial emphasis often placed on 
schedule 

■ Drives shortcuts 

■ Scope changes 

■ “Desirements” creep in 

■ Change orders required 

■ Initial assumptions were optimistic & 
incorrect 

■ Project not adequately planned due to 
schedule pressure 

■ Zero cost change orders 

■ Funding doesn’t show up on time, 
or is cut 

■ Big problem on international projects 

■ Unintended consequences 

■ the ripple effect 

■ Lack of cost and schedule controls 
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■ True requirements not clearly defined 

■ Risks not identified 

nsufficient contingency 

Ignore/Influence Independent estimates 


Top Reasons Reality Exceeds Initial Projections 


Escalation Assumptions Should be Realistic 
NASA New Start Escalation Index 





“The current efforts to help revive the economy are likely to produce inflation 
that could be worse than what the country suffered in the late 1970s.” 

Warren Buffett, March 2009 
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The Cost Of Change 

Nothing is Free 

□ Every document, change, report, study, review, test, analysis, conference, 
telephone call, etc., has a cost in manpower & material. All too frequently 
managers will delude themselves into thinking an action has "no cost" or 
"no impact" when the true assessment would be that the action can be 
undertaken with a relatively small effort. Also, decisions to change an item are 
frequently made without a thought-out assessment of all aspects of the task & 
with only an assessment of the visible work involved. 

□ For example, when workshop was in the final stages of preparation, a number of 
decals (containing operating instructions) were judged to be less clear than 
was possible. When the final bill was known, each decal change cost tens of 
thousands of dollars, all accountable**. The visible task, removal of the old 
decal, the preparation & installation of the new, was only a few dollars, as you 
would suspect. However, good aerospace procedures were used & work orders 
had to be prepared & processed, the work had to be reviewed & inspected, the 
drawings had to be changed, the change had to be processed, drawing lists had 
to be modified, test procedures modified & reviewed, crew checklists had to be 
changed, & to all this, overhead costs had to be added. 

□ Changing options, may reduce cost, but a price must be paid for everything. 

□ SKYLAB LESSONS LEARNED AS APPLICABLE TO A LARGE SPACE STATION, A dissertation 
submitted to the faculty of The School of Engineering & Architecture Of the Catholic University of 
America For the Degree Doctor of Engineering by William C. Schneider, Washington, D.C., 1976. 



**Sticker changes cost in excess of 100K each in FY11 $ 
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Don’t Assume All Contractors Are Honest 


Northrop Settles False Claims Case 

Northrop Grumman Corp. has agreed to pay $12.5 million to federal 
government to settle allegations that it submitted false claims to 

government agencies in connection with electronic parts it supplied for 
navigation systems for military airplanes, helicopters and submarines. 

Northrop's accused of failing to test commercial-grade electronic 
components to ensure they would function at extreme temperatures 
required for military & space uses. 

□ Number of DOD actions for contracting fraud FY01-FY10 

□ SAIC 139,331 

□ Lockheed Martin 119,499 

□ Northrop Grumman 99,454 

□ Raytheon 76,515 

□ General Dynamics 68,339 

□ Boeing 51,856 Actions 

□ Honeywell 55,078 
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“Rogue Contractors” 


□ Pentagon Paid $998,798 to Ship Two 19-Cent Washers 

□ 2007 - Small South Carolina parts supplier collected ~$20.5 million over six 
years for fraudulent shipping costs, including 

$998,798 to ship two 19-cent washers to Fort Bliss (Army base in Texas) 
$455,009 to ship three machine screws costing $1.31 ea. to Habbaniyah, Iraq 
$293,451 to ship 89-cent split washer to Patrick Air Force Base, Florida 

□ Exploited flaw in automated DOD purchasing system: 

Shipping bills to combat areas or U.S. bases labeled “priority” usually paid 
automatically 

□ C&D & 2 of its officials barred in December from receiving federal contracts 

Charlene Corley, 46, was fined $750,000. 

Faces maximum prison sentence of 20 years on each count 

□ Fraudulent billing started in 2000 

Started small & got more aggressive over time when invoices not questioned 
Purchase price of items rarely reached $100 & totaled ~$68,000 over six years 
Total shipping charges paid were $20.5 million 

□ Scheme Detected 

In September after purchasing agent noticed $969,000 bill for shipping two 19- 
cent washers 
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Confirmation Bias 


□ Philosopher Francis Bacon in 1620 

□ “the human understanding, once it has adopted an opinion, collects 
any instances that confirm it, & though the contrary instances 
may be more numerous & more weighty, it either does not 
notice them or else rejects them, in order that this opinion will 
remain unshaken” 

□ People have an unjustified bias in favor of their opinion 

n Much tougher when critiquing validity of information that undercuts one of 
their currently held theories than they are in supporting information that 
apparently endorses one of their own tenets. 

n Employing this double standard, tend to dismiss information that doesn’t fit 
with what they already believe. 
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Stating the Obvious 


□ Estimates only as good as inputs provided 

□ In 2008 MIT surveyed 80 systems engineers 

They were asked a series of questions that included 

□ How many countries have McDonalds? 

□ What is range of a Minuteman Missile in miles? 

□ How long in minutes & seconds is song Stop in the Name of Love? 

□ How many rulers has England had in the last 1,000 years? 

□ How tall is Sears Tower in meters? 

Asked large enough range to provide 90% certainty answer correct 

□ Example - 1 am 90% sure that there are between 50 & 150 countries with McDonalds 

• 83% of Systems Engineers got 50% or less answers correct. 

- Worse in 221 times experts said they were 100% sure, only correct only 73% of the time 

□ 1999 study of 27,000 expert predictions - did not vindicate their "expertness" 
as anticipated. 

□ Error rates many times what experts predicted. 

□ No advantage for experts holding advanced degrees. 

• Why is this important? 

• If a system engineer tells you they are 90% certain about something - They 
will be WRONG much of the time!! 

• Base as many inputs as possible on objective data - not opinions 


“In god we trust; all others must bring data ” Edward Deming 



Conclusions 



□ Presentation probably not what you expected 

□ Before you can get help you have to admit there is a problem 

□ My crystal ball says there is a high probability that SKA will 
overrun current cost and schedule budgets 

□ I strongly suggest that project 

□ Build in options that can be easily cut, or done later, when problems arise 

□ Do one or more independent assessments that are truly independent 

□ Have strong project management team with business background 
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Backup 
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The 

Joint Confidence Level Paradox: 
- A History of Denial - 

2009 

NASA Cost Symposium 
(Tuesday, April 28th) 



PRESENTERS: 

Glenn Butts: CCC, MRICS, CGC, CMC, CFC 

Kent Linton: EVMS & Cost Estimating - SAIC/Craig Technologies 


□ If you are interested in this 
topic, I wrote a 1 00 page 
paper on the subject 

□ Email me for copy 

□ Glenn.C.Butts@nasa.gov 
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We Claim to Seek Accurate Estimates 


□ Yet, we; 

□ Persist in submitting optimistic initial estimates 

□ Omit probable scope from our estimates that often become "real." 

□ Exclude credible risks from our analysis 

SM&A actively mitigates risks with a likelihood of occurrence of only one in several 
thousand, & are supported by management. 

□ Management mantra 

□ “if status quo is assumed, we will get the status quo” 

Meaning status quo is too expensive 

There are valid reasons for why things cost what they cost 

Excuse for under-estimating projects 

□ Possible conclusions: 

□ Accurate estimates are impossible, especially at early stages 

The variables are too complex, unquantifiable, & incalculable - NOT 

□ We really don’t want accurate estimates; 

Primary concern is keeping program viable & funded as long as possible 
Early estimates are deliberately misrepresented & meant to gain approval 
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How Do We Underestimate? 

- Let Me Count The Ways - 



1 . OMIT PROBABLE SCOPE from estimate 

2. OMIT POSSIBLE RISKS from analysis 



Rand Study - Reason 
for 74% of Cost Growth 


□ Internal & External 


3. UNREALISTIC, OPTIMISTIC assumptions I 

4. Use historically LOW ESCALATION projections 

□ Rand Study - Reason for 1 1 .2% of Cost Growth 

5. Issue cost estimates in BASE YEAR dollars 

□ Estimates should be in then year dollars (escalated to year in which it is spent) 

6. Many estimates NOT PREPARED BY A BONA FIDE ESTIMATOR 

□ Everyone's a estimator 

□ Being certified no guarantee of having necessary experience 

7. REWARD failure, PUNISH honesty 

□ In my opinion many managers don’t want good estimates 

8. NOT ENOUGH TIME to prepare CREDIBLE estimates 

□ Time often spent doing “what if exercises, or splitting dollars into arbitrary buckets 

□ Spend more time in reviews & defending estimates than preparing them 


“I reject a system that rewards failure & protects a 
person from its consequences” 


- Barack Obmno - 
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Early Estimates are Often Deliberate Misrepresentation 

Polite way of Saying Lying 



□ Conclusions from the largest study of its kind: 

□ Examined 258 large transportation projects 

□ Underestimation has not decreased during past 70 yrs. 

□ It was found with overwhelming statistical significance that 
cost underestimation cannot be explained by error. 

“Is best explained by strategic misrepresentation, namely lying , 
with a view to getting projects started.” 

□ Cost overruns of 50% to 100% in real terms are common. 

Overruns above 100% are not uncommon. 

□ Accuracy is gained at each successive stage of progress. 

□ Promoters routinely ignore, hide, or otherwise leave out 
important project costs & risks in order to make total costs 
appear low. 

• Cost underestimation occurred in almost 9 out of 10 projects. 

- The likelihood of actual costs being larger than estimated is 86% 

- The likelihood of actual costs being lower than estimated is 14% 



This study looked at comparatively simple projects. 
Statistics for more complex projects would likely be worse! 
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Final Overrun (%) to Commitment 
at Start of Development 


Planning Stage 

We don’t adequately study project before we commit. 
Don’t want to “waste” money on studies. 



180 


160 



Study Phase Cost as a Percent 
of Development Cost 


Spending More 
Money During 
Early Phase = 
Substantially Less 
Cost Growth 
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International Space Station 

□ 1984 President Reagan directed NASA to build a space station 
within a decade and to invite other countries to join the endeavor. 

□ 1991 the US House of Representatives held the first of 22 separate 
votes on whether to proceed with the program. 

□ As a result of overruns it came within 1 vote of cancellation in 1 993. 



□ “For years the agencies cost overruns and schedule delays 
supplied ammunition for annual congressional attempts to kill the 
International Space Station” - Florida Today 
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Multiple International Partners Adds Complexity 


■ The more partners the more risk is added to the project. 

■ If one partner has difficulties it often has ripple affects 

■ ESA's way of doing business is even less efficient than NASA’s 
because they have the added complexity of making sure on any given 
project all 18 member nations puts in funding for a given project that 
matches their percentage "ownership" in ESA. 

■ This can often make their contracting more difficult because they just 
can't let Company X in Country A do something that they are most 
efficient at and instead have to let the work be done by Company Y in 
Country B to keep the percentages correct. 
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International Partners 


□ TheISS 

□ “The U.S. developmental effort cannot be isolated from these occurrences and their 
associated impacts: the Program has experienced cost growth and schedule slippage 
associated with this broad level of international involvement.” Source 

http://historv.nasa.gov/32999.pdf 

□ Another ISS resource about international partner risks - see Appendix C, International 
Participation. From http://oiq.nasa.aov/conqressional/testimonv040401.doc 

□ The ITER project 

□ “The expert panels identified a number of factors arising from significant changes in 
circumstances and the parameters of the work necessary and contributing to the increases 
of cost estimates between 2001 and 2008. The most important factors relate to: increased 
complexity of project integration in managing interfaces across seven ITER Parties (in 2001 
only 3 Parties were foreseen to execute the project)” Source 
http://fire.pppl.gov/iter europa cost 050610.pdf 

□ GAO report. From http://www.qao.gov/new.items/d09306sp.pdf 

□ “These partnerships go a long way to foster international cooperation in space, but they 
also put NASA projects in a vulnerable position when partners do not meet their obligations 
or run into technical obstacles they cannot easily overcome.” 

□ “Project officials also commented that Aquarius had changes to its baseline due to slips by 

its development partner, the CONAE, [Argentina] and that they believe the NASA 
contribution to this mission is on schedule for completion in March 2009. They added that 
the benefit of the international partnership, plus the groundbreaking information about the 
Earth’s climate, out weigh the additional costs, which NASA has chosen to absorb within its 
budget. 3£ 


Report to the Committee on Science, House of Representatives 

May 2004 

GAO-04-642 Report @ http//www.gao.gov/cgi-bin/getrpt?GAO-04-642 

□ “Lack of Disciplined Cost-Estimating Processes Hinders Effective 
Program Management” 

□ NASA lacks clear understanding of programs cost & schedules. 

Estimates for more than half of programs reviewed have increased — as much as 
94 percent. 

Cost Estimates increased 80% of time 

Variability in cost estimates indicates programs lacked sufficient knowledge 
needed to establish priorities, quantify risks, & make informed investment 
decisions, & thus predict costs. 

□ “NASA’s basic cost-estimating processes — an important tool for 
managing programs — lack discipline needed to ensure that 
estimates are reasonable.” 

□ None of programs reviewed in detail met all cost-estimating criteria. 

□ “NASA has limited ability to collect program cost & schedule data 
needed to meet basic cost-estimating criteria.” 

□ NASA does not have a system to capture reliable financial & performance data — 
key to using effectively the cost-estimating tools that NASA officials state that 
programs employ . 

□ NASA identified other barriers, including limited cost-estimating 
staff. 
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GAO Cost Estimating Criteria 



1) Objectives of estimate shall be 
stated in writing. 

2) Life cycle to which estimate 
applies is clearly defined. 

3) Tasks appropriately sized. 

4) Estimated cost & schedule are 
consistent with demonstrated 
accomplishments of other 
projects. 

5) Written summary of parameter 
values & their rationales 
accompany estimate. 

6) Assumptions shall be identified 
& explained. 

7) Structured process such as a 
template or format shall be used 
to ensure that key factors are not 
overlooked. 


8) Uncertainties in parameter values 
are identified & quantified. 

9) If schedule is imposed, estimate 
for normal schedule shall be 
compared to additional cost 
required to meet imposed 
schedule. 

10) If more than one estimating 
approach is used, variances are 
analyzed & explained. 

1 1 ) Independent assessment concur 
with the values & methodology. 

12) Estimates kept current. 

13) Estimates integrated with project 
planning & tracking. 

14) EVM is used to manage program. 


Basically - Good Processes & Documentation 
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Underestimates are Nothing New - Quotes from 1964 

All but One from Apollo Executives Meeting 


□ 7 can think of a lot of programs in the Boeing Company where, if the estimate 
had been realistic , you wouldn't have had the program , & that is the truth." 

W. M. Allen - President, Boeing 

□ "You have basically built into the entire system a series of events which 
very much leads to under-estimating what the program is going to cost" 

J. S. Parker - Vice President, GE 

□ "As advocates of new programs , government agencies have often 
encouraged contractors to estimate costs optimistically. 

□ Recognizing that headquarters might be shocked out of supporting a program 
where true costs were revealed at the outset. 

□ They have sought to disclose cost increases gradually, after programs have 
gained momentum & cancellation has become difficult. " 

Dr. Frederic Scherer - Harvard 

□ "We can give you estimates all the way to the end of the Saturn-IVB, but do 
you really want to know it?" 

Donald W Douglas - President Douglas Aircraft 



Underestimates documented back to 1869 - probably prior. 
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Common Causes of Cost Problems 


□ Optimistic assumptions 

□ Poor estimating techniques/standards/processes 

□ Inadequate work breakdown structure 

□ Technology readiness 

□ Managerial edicts to reduce estimates by eliminating “fat” 

□ Inadequate planning that results in increased scope 

□ Requirements creep 

□ Unforeseen technical problems 

□ Schedule delays 

□ Inadequate workforce skill mix 

□ Contractor buy in 

□ Lack of historical data 
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Common Causes of Cost Overruns 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 



Failure to understand customer requirements 
Unrealistic assessment of current capabilities 
Underestimating time requirements 
Omissions (inadvertent or deliberate) 

Misinterpretation of information 
Inappropriate estimating techniques 
Failure to identify & concentrate on major cost elements 
Failure to recognize & plan for risks 
□ Ignore external risks 
Pressure to compromise early 
Ceiling Costs 

Must win mentality (Success Oriented) 

Poorly written contractual requirements 
Proposal team different from Project team 
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Forgotten SPACE SHUTTLE Facts 




□ Initially approved by 1 Vote 

□ Congress WAS concerned about estimates 

□ FY 1972 Estimate 

□ DDT&E $5.5 Billion + 20% Contingency = $6.6 Billion 

Contingency Reportedly Removed by Congress 

□ Facilities $0.3 Billion 

□ 1 st Manned Flight projected by June 1976 (4 yrs.) 

□ Reality -1982 

□ Total Cost $17,789 Billion 

DDT&E $13,138 Billion 
Production $ 3.900 Billion 
Facilities $ 0.703 Billion 

□ 1 st Manned flight occurred in April 1981 

□ Increase WAS Foreseen . . . 

□ “Considering all of the technological & operational unknowns involved in the shuttle 
development & the fact that no vehicles of similar function have ever been designed before or 
have ever operated over the range of flight regimes required for the shuttle, prudent 
extrapolation of prior experience would indicate that estimated development costs may be 30 
to 50 percent on the low side. Thus, the estimates of $6. 5 billion in RDT&E for the Mk l/Mk II 
shuttle program may range between $8.5 to $10 billion, reflecting increased program costs of 
$2.5 to $3.5 billion. Similar uncertainties must be considered to apply to other non-recurring 
costs such as production & facilities (amounting to about $4 billion). Thus a possible cost 
uncertainty of about $5 billion for total program costs might be envisioned giving a high 
estimate of total non-recurring cost of about $15 billion . ” - October 1971 Report - 


“The chance of a Shuttle launch on 
November 20 1980 is probably 10 to 20%” 

Dr. Frosch NASA - Administrator 
Congressional Testimony 1980 

(9 yrs.) 
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Were to Start? - Elements That Influence Costs 




□ What - is project scope? 

□ What are boundaries of work 

□ Who will do required work outside 
boundaries? 

□ Who - will do 

□ Design? 

Self 

On call A&E 
New A&E 

On board Support contractors 

□ Self perform work 

□ Sub out 

□ Work? - FFP or Cost Plus 

On board support contractors 

□ Self perform work 

□ Sub out 

General Contractor 

General Contractor & Sub Contractor’s 


□ How - will project be bid? 

□ Full & open 

□ Source board 

□ IDIQ 

□ Sole source 

8a - Disadvantaged 
Support contractor 
Other 

□ Best value 

□ Design build 

□ Cost plus 

□ Where - will project be done 

□ On center 

□ Off center 

Costs may change with location change 

□ When - will the project be done 

□ Required for escalation calculations 

□ Anticipated market conditions 

Negative or positive 


Point 

Details Matter! 
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Backup 

Misc Projects Lessons 

Learned 


47 



A 


1 . 


2 . 


3 . 


4 . 


5 . 

6 . 


7 . 


8 . 


CLCS Summary - Lessons Learned 
$400 M Software Project 



Independent Project Oversight 

• Vigorous independent review process critical to ensure early warning to senior management. 

• Senior management must follow-up on independent reviews & control performance 

Control Project Requirements 

• Control requirements early & ensure understanding of them between stakeholders 

• Must be placed under configuration control - with no changes unless through analysis is done 

• Any changes must be carefully balanced by cost, development & operational risks 

Risk Management Mitigation 


Huge 

Overruns 

Project 

Canceled 


Clear identification of each significant risk, their probability of occurrence, impact of risks on project, & clear plan for risk 
mitigation are critical to project success 

Cultural impediment to implementation are significant and must be overcome 
Implement periodic validation of risk assessment by group external to the project 


System Integration & Performance Management 

• Implement an effective Systems Engineering group to be responsible & accountable for systems design, integration, 
performance analysis & verification 

• Lack of early planning manifests itself in cost overruns, failed acceptance testing, slipped schedules, and disruption of planned 
integration of interfacing systems. 

Milestones 


• Project milestones must explicitly defines what capabilities are included in each 

Communications 

• Project managers must focus on meeting approved requirements while controlling cost & schedule. 

• Communication, requirement definition & integration should be prime management metric. 

Estimating 

• Sound cost & schedule estimating early, including a bottoms-up cost estimate, coupled with use of accurate performance 
management system is essential to meeting project goals. 

Contract Vehicle 

• Contracts should be structured such that performance on important work has an overall effect on performance award. 

• Caution should be utilized such that the conflict of interests are created 


NASA continues to relearn lessons already learned from the past. 


A3 Lessons Learned 


□ Phased Funding 

□ Not delivered when promised 

Drove segmenting of bid packages 

□ Budget cut 

Have de-scope options planned Vi SKA? 

□ Requirements 

□ Geographically separated teams maintain separate requirements documents 

Drove Change Orders & Schedule slips 

□ LOX barges pumps changed, new requirement for VFDs. Great impact to electrical power systems 
(requiring larger diameter wiring, conduit, etc). However, decision NOT coordinated with A-3 project. 

□ All not available until design 30% complete - Impact cost and schedule 

□ Concept, schedule & budget locked before all requirements known 

Initial concept did NOT include plans for buildings to house electronic equipment required for 
facility control & capturing test data. 

Planned availability dates for necessary buildings not incorporated into overall project 
schedule making it extremely difficult to plan for installation, verification, and activation tasks 

□ Not reviewed by ALL the necessary stakeholders 

□ All “rated” equally - some critical, some are not - all could not be met 

Vendors unaware of importance did not meet some requirements with proposals 

□ Project data maintained on three different database systems 

Everyone does not have access to all systems, so data duplicated & not always updated 
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A3 Lessons Learned 


□ Design changes 

□ Result of review comments and resulting in scope growth/cost growth 

□ Design schedule difficult to maintain with changing scope 

□ Cryogenic tanks resized due to high bids - driven by transportation costs 

Drove changes to other systems 

Original size tanks could have been shipped by barges, not investigated 

□ Design errors / conflicting data 

Driven in part by schedule, in part by late stage design changes 

□ Rush to maintain schedule/budget 

□ Required contract awards before design complete - caused cost of some contracts to 
double 

□ Defer factory work to field, where skills not always available to perform 

□ Company who will install items did not order them 

When they arrived late, installation contractor got change order 
Another company responsible for transportation 

□ Preliminary System Hazard Analysis was not performed in concept phase 

Drove later changes to mitigate hazards 

□ Short timeline on requests for budget / schedule actions 

Does not provide sufficient time to properly evaluate requirements & formulate plan 

□ Example - request to provide estimate for Low Speed Video System in one day 

□ Subsequent costs were higher, and additional funding difficult to obtain 
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A3 Lessons Learned 


□ Bureaucracy 

□ Nine months to order equipment, delivery only took 6-8 weeks after order finally placed 

This occurred on small orders $150K & less 

Small contracts should be treated differently than big ones 

□ Excessive regular meetings that project consume time which could be applied to productive 
work 

Three to six meetings a week in various forums, does not allow sufficient time to perform project 
actions 

□ Failure to utilizing remote meeting capability effectively 

It good to have face-to-face meetings yet as a project progresses, travel times for meetings can 
impact delivery schedules 

Use of internet format meetings allows sharing of technical documents with minimal interruptions to 
productive work 

□ Regulations on contract value limits drove splitting of bid packages into small packages 

Increased confusion 

Increase coordination work on project team 

□ Risks 

□ Optimistic initial estimates by people with vested interest in gaining project approval 

□ Failure to accept independent cost assessment 

Exerted pressure to force retraction and agreement with projects number 

□ Hurricane Katarina 

Due to rebuilding efforts costs were substantially higher than estimated 

□ Other 

□ Coordination of multiple contracts on scope & availability sometimes a problem 

If project was one large fixed price contract you know who has scope 
□ Items in design don’t get lost in the shuffle 
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Lessons Learned 




□ Project Formulation 

□ Watch out for unrealistic cost and schedule estimates resulting from: 

□ Inaccurate/incomplete understanding of requirements for spaceflight end items 

□ “low balling” in order to get an ATP 

□ an overrating of the technology maturity 

□ Procurement 

□ Never agree to a cost plus fixed fee contract for a research, design 
and development effort 

□ The contractor bears no risk -- it’s all on the government 

□ The government has no leverage with the contractor -- no incentives (positive or 
negative) for addressing problems 


Lessons Learned 


□ Project management 

□ Contractor Project Manager should have a proven track record (deliveries, 
communications, risk management) 

□ Don’t skimp on oversight functions - match each major contract 
skill/function with an accountable government counterpart 

□ You must have reliable, accurate, and timely cost reporting 

□ Pay close attention to environments: 

□ Perform functional testing in relevant environment to the extent possible 

□ For situations you can’t thoroughly test for, do “what if study to find 
possible effects and solutions 

□ Don’t ignore quirky test data -- it may be telling you something important 

□ Develop back up plans for all reasonably possible contingencies 

□ Conduct comprehensive failure or anomaly investigations 

□ Take whatever action is necessary to control the contractor -- be prepared 
to descope or terminate 

□ Take risks for good reasons and with management's buy-in 

□ Trust dedicated, “can do” team members to come up with creative solutions 


j. ~ : 


